Apparatus and method for combining images

ABSTRACT

Provided are an image composition apparatus for composing color images with black-and-white images including infrared components, and an image composition method thereof. The image composition method includes generating a first image signal with color information and a second image signal including infrared components without color information, dividing the first image signal into a brightness signal and a color signal, composing the brightness signal of the first image signal with a brightness signal of the second image signal to generate a composed brightness signal, and composing the composed brightness signal with the color signal of the first image signal to generate a color image.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation application of prior application Ser.No. 15/676,235, filed on Aug. 14, 2017, which is a continuationapplication of prior application Ser. No. 14/627,498, filed on Feb. 20,2015, which has issued as U.S. Pat. No. 9,736,403 on Aug. 15, 2017,which is a continuation application of prior application Ser. No.12/467,321, filed on May 18, 2009, which has issued as U.S. Pat. No.8,989,487 on Mar. 24, 2015 and claimed the benefit under 35 U.S.C. §119(a) of Korean Patent Application number 10-2008-0046060, filed on May19, 2008, the disclosure of which is incorporated by reference herein inits entirety.

BACKGROUND Field

The following description relates to a technology of combining images,and more particularly, to an apparatus and method for combining colorimages with black-and-white images including infrared components.

Description of the Related Art

With popularization of digital cameras, interests in digital imagingdevices have been growing. A digital imaging device may edit or storecaptured images as it digitalizes and processes various imageinformation.

In general, a digital imaging device includes a lens, an image sensor,and an image processor. The lens adjusts a focus of light reflected froman object and transmits the light to the image sensor so that the lightforms a proper image on the image sensor. The image sensor senses thelight incident thereon and generates image signals, that is, electricalsignals. The generated image signals are subjected to processing and maybe displayed or stored.

Types of image sensors include image pickup tubes and solid imagesensors. The solid image sensors include charge coupled devices (CCDs),complementary metal-oxide-semiconductors (CMOSs), and the like.

A CCD sensor includes a circuit in which a plurality of capacitors areconnected in pairs. Also, a CCD chip including a plurality of photodiodes generates electrons according to an amount of light incident oneach photodiode. Then, by reconfiguring information generated by thephotodiodes, image information may be created.

CMOS image sensors may be manufactured at lower costs than CCD imagesensors as the CMOS image sensors may be manufactured using ageneral-purpose semiconductor manufacturing equipment. Therefore, CMOSimage sensors have been typically utilized in low-priced digital camerasor slow-frame television cameras. However, CMOS image sensors may beunstable or have poor performance in a low illumination environment, andimages captured by a CMOS image sensor may have noises.

While a CMOS image sensor can convert infrared light as well as visiblelight into image signals, such infrared components are generally removedby an infrared blocking filter in order to easily restore color signals.However, in order to acquire images over wider bands, it is desirable touse the infrared components.

SUMMARY

According to one general aspect, there is provided an image compositionapparatus including an image acquiring unit to sense incident light andgenerate a first image signal with color information and a second imagesignal including infrared components without color information, an imagesignal divider to divide the first image signal into a brightness signaland a color signal, a brightness composer to compose the brightnesssignal of the first image signal with a brightness signal of the secondimage signal to generate a composed brightness signal, and an imagerestoring unit to compose the composed brightness signal with the colorsignal of the first image signal, so as to generate a color image.

The first image signal may include a signal corresponding to a specificregion of a visible band of an optical spectrum, and the second imagesignal may include a signal corresponding to an infrared band of theoptical spectrum and a combination of signals corresponding to specificregions of the visible band.

The first image signal may be a color image signal, and the second imagesignal may be a black-and-white image signal including infraredcomponents.

The apparatus may further include a color space converter to convert acolor space of the first image signal.

The apparatus may further include a dynamic bandwidth adjusting unit toequalize dynamic bandwidths of the first image signal and the secondimage signal.

The apparatus may further include a resolution adjusting unit toequalize resolutions of the first image signal and the second imagesignal.

The apparatus may further include a domain transformer to transformspatial domains of the first image signal and the second image signalinto frequency domains.

According to another aspect, there is provided an image compositionmethod in an image composition apparatus, the method includinggenerating a first image signal with color information and a secondimage signal including infrared components without color information,dividing the first image signal into a brightness signal and a colorsignal, composing the brightness signal of the first image signal with abrightness signal of the second image signal to generate a composedbrightness signal, and composing the composed brightness signal with thecolor signal of the first image signal to generate a color image.

The method may further include converting a color space of the firstimage signal prior to the dividing of the first image signal into thebrightness signal and the color signal.

The method may further include equalizing dynamic bandwidths of thefirst image signal and the second image signal. The equalizing of thedynamic bandwidths of the first and second image signals may comprisecompressing the second image signal to match with the dynamic bandwidthof the first image signal.

The method may further include matching a resolution of the first imagesignal with a resolution of the second image signal. The matching of theresolution may comprise interpolating an image signal with the lowerresolution among the first and second image signals with respect to animage signal with the higher resolution among the first and second imagesignals.

Other features and aspects will be apparent from the following detaileddescription, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an exemplary image compositionapparatus.

FIG. 2 is a diagram illustrating an optical spectrum.

FIG. 3 is a diagram illustrating a configuration of an exemplary imageacquiring unit.

FIG. 4 is a block diagram illustrating another exemplary imagecomposition apparatus.

FIGS. 5 through 8 are block diagrams illustrating still anotherexemplary image composition apparatuses.

FIG. 9 is a flowchart of an exemplary image composition method.

FIG. 10 is a flowchart of another exemplary image composition method.

Throughout the drawings and the detailed description, unless otherwisedescribed, the same drawing reference numerals will be understood torefer to the same elements, features, and structures. The relative sizeand depiction of these elements may be exaggerated for clarity,illustration, and convenience.

DETAILED DESCRIPTION

The following detailed description is provided to assist the reader ingaining a comprehensive understanding of the methods, apparatuses,and/or systems described herein. Accordingly, various changes,modifications, and equivalents of the systems, apparatuses and/ormethods described herein will be suggested to those of ordinary skill inthe art. Also, descriptions of well-known functions and constructionsmay be omitted for increased clarity and conciseness.

FIG. 1 shows an exemplary image composition apparatus.

The image composition apparatus may be applied to an imaging device todetect light reflected from an object and create or store an image. Forexample, the image composition apparatus may be applied to a digitalcamera, a hardware system to drive a digital camera, an image processingchip, and the like. Referring to FIG. 1, the image composition apparatusincludes an image acquiring unit 101, a color space converter 104, animage signal divider 105, a brightness composer 108, and an imagerestoring unit 109.

The image acquiring unit 101 may be a CCD or CMOS image sensor to senselight reflected from an object and converting the sensed light intoelectrical signals. The image acquiring unit 101 senses light incidentthereon to generate a predetermined image signal, wherein the imagesignal may be divided into a first image signal 102 and a second imagesignal 103.

The first image signal 102 includes color information, and the secondimage signal 103 includes infrared components without any colorinformation. The first and second image signals 102 and 103 will bedescribed further with reference to an optical spectrum illustrated inFIG. 2.

Referring to FIG. 2, light may be classified into a visible band 111, aninfrared band 112 and an ultraviolet band 113 according to itswavelength. The first and second image signals 102 and 103 may meansignals belonging to specific wavelength bands of an optical spectrum.For example, the first image signal 102 includes image signalscorresponding to light belonging to specific regions 114 of the visibleband of the optical spectrum, and the second image signal 103 includescombinations (for example, 300) of signals belonging to the specificregions 114 of the visible band 111, and image signals corresponding tolight belonging to the infrared band 112 of the optical spectrum. Inother words, the first image signal 102 is a color image signal withcolor information, and the second image signal 103 is a black-and-whiteimage signal including infrared components without color information.

Image signals, such as the first and second image signals 102 and 103,having information of different wavelength bands, may be obtained byusing, for example, a multi-sensor technology utilizing differentoptical systems and image sensors or by controlling the filteringfunction of a color filter array (CFA) without having to change anoptical system.

FIG. 3 illustrates a configuration of an exemplary image acquiring unit,wherein the image acquiring unit may be used to generate the first andsecond image signals 102 and 103 described above.

Referring to FIG. 3, the image acquiring unit includes an image sensor116, and a filter array 115 disposed on the image sensor 116 and coupledwith the image sensor 116. The image sensor 116 senses light incidentthereon and converts the light into electrical signals, and the filterarray 115 filters light that is to be incident on the image sensor 116so that light belonging to specific wavelength bands is only incident tothe image sensor 116.

The filter array 115 includes color filters 117 to selectively transmitlight belonging to specific regions (for example, the regions 114 ofFIG. 2) of the visible band of an optical spectrum, and a transparentfilter 118 to transmit light over all bands of the optical spectrum. Thefilter array 115 has no infrared blocking function.

Accordingly, where light reflected from an object passes through thefilter array 115, the color filter units 117 transmit light belonging tospecific bands of the visible band and infrared light therethrough, andthe transparent filter 118 transmits light (including infrared light)over all bands therethrough.

The image sensor 116 disposed below the filter array 115 may have amulti-layer structure with stacked sensor modules. For example, theupper layer 130 of the image sensor 116 includes first light receivers119 to sense light belonging to the visible band from the light whichhas passed through the color filters 117, and a second light receiver120 to sense white light which has passed through the transparent filter118, and the lower layer 131 of the image sensor 116 includes a thirdlight receiver 121 to sense light belonging to the infrared band.

The image sensor 116 may be formed by a semiconductor manufacturingprocess, and each light receiver may be a photodiode made of silicon.Here, since infrared light with wavelengths longer than those of visiblelight is absorbed at a relatively deeper location (that is, the lowerlayer 131), the multi-layer structure is provided in which the upperlayer 130 detects light of the visible band and the lower layer 131detects light of the infrared band.

In the image acquiring unit, the first light receiver 119 detects lightwith color information, the second light receiver 120 detects light (forexample, white light) without color information, and the third lightreceiver 121 detects infrared light, respectively. Accordingly, theoutput signal of the first light receiver 119 is used as the first imagesignal 102, and the output signals of the second and third lightreceivers 120 and 121 are used as the second image signal 103.

Referring back to FIG. 1, the color space converter 104 converts a colorspace of the first image signal 102. As described above, since the firstimage signal 102 has color information, the first image signal 102 maybe represented as an RGB signal in a color space. For example, the colorspace converter 104 may convert the first image signal 102 representedas an RGB signal into a YCbCr signal, using a color space conversionfunction. This is only one example and the color spacer converter 104may convert the RGB signal into another signal, such as HSV, HIS, LUV,and the like.

The first image signal 102 whose color space has been converted by thecolor space converter 104 is input to the image signal divider 105 (seeFIG. 1), and the image signal divider 105 divides the first image signal102 into a brightness signal and a color signal 107.

Here, the brightness signal 106 corresponds to brightness informationfor the first image signal 102, and the color signal 107 corresponds tocolor information for the first image signal 102. For example, if thefirst image signal 102 represented as an RGB signal is converted into aYCbCr signal, the brightness signal 106 corresponds to the Y signal andthe color signal 107 corresponds to the CbCr signal. Also, in a HSVspace, V information may be used as the brightness signal 106, and in aHIS space, I information may be used as the brightness signal 106.

The brightness composer 108 composes the brightness signal 106 dividedfrom the first image signal 102 with the brightness signal of the secondimage signal 103. Since the second image signal 103 has only brightnessinformation without any color information, the second image signal 103may be composed with the brightness signal 106 divided from the firstimage signal 102 without any additional processing. For example, if thesecond image signal 103 is represented in the YCbCr color space, the Yinformation of the second image signal 103 is composed with the Yinformation of the first image signal 102 output from the image signaldivider 105.

A brightness composition method may be used in which brightness signalsare composed using, for example, a lookup table or a conversion functionrepresenting the relationship between the brightness information of thefirst image signal 102 and the brightness information of the secondimage signal 103. As another example, a brightness composition methodmay be used which adds the coefficients of brightness signals usingDiscrete Wavelet Transform (DWT).

The image restoring unit 109 combines the composed brightness signal 110generated by the brightness composer 108 with the color signal 107 ofthe first image signal 102 divided by the image signal divider 105, soas to generate a color image. Combining the composed brightness signal110 with the color signal 107 may be done by the inverse processing ofthe division processing by the image signal divider 105, and the finalcolor image may be obtained by the inverse processing of the conversionprocessing by the color space converter 104.

Accordingly, the color image provided by the image composition apparatusis an image created using both visible light signals and black-and-whiteimage signals including infrared components. That is, by using signalsover a wide band, recognizable image information may be obtained even ina low illumination environment.

FIG. 4 shows another exemplary image composition apparatus. Here, theimage composition apparatus adjusts the dynamic bandwidths of imagescompared to the image composition apparatus of FIG. 1.

Referring to FIG. 4, the image composition apparatus includes a dynamicbandwidth adjusting unit 122. The second image signal 103 whichcorresponds to a black-and-white image may include image informationwith a bandwidth wider than that of the first image signal 102 whichcorresponds to a color image. In this case, due to the second imagesignal 103, an unnatural image may be created upon image composition.

The dynamic bandwidth adjusting unit 122, which equalizes the dynamicbandwidths of the first and second images signals 102 and 103, adjuststhe dynamic bandwidth of the second image signal 103 to be suitable foran image output apparatus, or compresses the dynamic bandwidth of thesecond image signal 103 to be equalized to the dynamic bandwidth of thefirst image signal 102.

FIGS. 5 through 7 show still another exemplary image compositionapparatuses, wherein the image composition apparatuses adjustresolution.

Referring to FIGS. 5 through 7, the image composition apparatusesinclude a resolution adjusting unit 123.

For example, the resolution adjusting unit 123 interpolates an imagesignal with lower resolution to equalize the resolutions of imagesignals. This process may be performed after brightness/color division(FIG. 5) or before brightness/color division (FIG. 6). Where the firstimage signal 102 which corresponds to a color image has a resolutionlower than that of the second image signal 103, the resolution adjustingunit 123 equalizes the resolution of the first image signal 102 to theresolution of the second image signal 103. Where the first image signal102 which corresponds to a color image has a resolution higher than thatof the second image signal 103 which corresponds to a black-and-whiteimage, the resolutions of the first and second image signals 102 and 103may be equalized by interpolating the second image signal 103 with thelower resolution.

FIG. 8 shows still another exemplary image composition apparatus.

Referring to FIG. 8, the image composition apparatus includes a domaintransformer 201 and a domain inverse-transformer 202.

The domain transformer 201 transforms the spatial domains of the firstand second image signals 102 and 103. For example, the domaintransformer 201 transforms the spatial domains of the first and secondimage signals 102 and 103 into frequency domains. The domaininverse-transformer 202 inverse-transforms the frequency domains intothe original spatial domains before generating a final color image.

FIG. 9 shows a flowchart of an exemplary image composition method. Theimage composition method may be performed by an image compositionapparatus described above.

Referring to FIG. 9, a first image signal with color information and asecond image signal including infrared components without colorinformation are acquired by sensing incident light in operation S201.Here, the first image signal may be a color image signal, that is, asignal which corresponds to specific regions of the visible band of anoptical spectrum, and the second image signal may be a black-and-whiteimage signal including infrared components, that is, a signal whichcorresponds to a combination of all or specific regions of the visibleband, or a signal which corresponds to the infrared band. The first orsecond image signal may be generated by, for example, the imageacquiring unit of FIG. 3.

The color space of the first image signal is converted in operationS202. For example, the first image signal represented as an RGB signalis converted into a YCrCb signal.

In operation S203, the first image signal whose color space has beenconverted is divided into a brightness signal and a color signal. Forexample, the brightness signal corresponds to brightness informationrepresented as a Y signal and the color signal corresponds to colorinformation represented as a CbCr signal.

Where the first image signal is divided into the brightness signal andcolor signal, the brightness signal of the first image signal iscomposed with the brightness signal of the second image signal togenerate a composed brightness signal in operation S204. The brightnesscomposition may be carried out by adding the coefficients of brightnesssignals using DWT to obtain a composed brightness signal.

In operation S205, the composed brightness signal is composed with thecolor signal divided in the operation S203, so as to generate a colorimage. Here, the color image may be obtained by the inverse processingof the division processing performed in the operation S203.

FIG. 10 shows a flowchart of another exemplary image composition method,wherein the image composition method includes an operation of adjustingthe resolutions and dynamic bandwidths of images.

Referring to FIG. 10, first and second image signals are acquired inoperation S301, and equalizing the resolutions of the first and secondimage signals is performed in operation S302. The operation of adjustingthe resolutions of the first and second image signals may be carried outby interpolating an image signal with the lower resolution among thefirst and second image signals to equalize the resolutions of the firstand second image signals.

In operation S303, the first image signal is divided into a brightnesssignal and a color signal after adjusting the resolutions of the firstand second image signals.

In operation S304, the dynamic bandwidths of the first and second imagesignals are equalized. The operation of equalizing the bandwidths of thefirst and second image signals may be carried out by compressing thesecond image signal with the wider dynamic bandwidth to be matched withthe bandwidth of the first image signal or by adaptively matching thedynamic bandwidth of the first image signal with the dynamic bandwidthof the second image signal.

In operation S306, the brightness signal of the first image signal iscomposed with the brightness signal of the second image signal togenerate a composed brightness signal. The composed brightness signal iscomposed with the color signal of the first image signal to generate acolor image in operation S306.

According to examples described above, color images are combined withblack-and-white images including infrared components.

According to examples described above, since black-and-white imagesignals having the wider band as well as color image signals areadditionally used to generate images, high-sensitivity images may beobtained in a low illumination environment.

According to examples described above, since infrared signals having nocolor information may be represented as black-and-white images,appropriate composition of black-and-white images with color images mayobtain images with high-sensitivity and wide-bandwidth.

The methods described above may be recorded, stored, or fixed in one ormore computer-readable media that includes program instructions to beimplemented by a computer to cause a processor to execute or perform theprogram instructions. The media may also include, alone or incombination with the program instructions, data files, data structures,and the like. Examples of computer-readable media include magneticmedia, such as hard disks, floppy disks, and magnetic tape; opticalmedia such as CD ROM disks and DVDs; magneto-optical media, such asoptical disks; and hardware devices that are specially configured tostore and perform program instructions, such as read-only memory (ROM),random access memory (RAM), flash memory, and the like. Examples ofprogram instructions include machine code, such as produced by acompiler, and files containing higher level code that may be executed bythe computer using an interpreter. The described hardware devices may beconfigured to act as one or more software modules in order to performthe operations and methods described above, or vice versa.

A number of exemplary embodiments have been described above.Nevertheless, it will be understood that various modifications may bemade. For example, suitable results may be achieved if the describedtechniques are performed in a different order and/or if components in adescribed system, architecture, device, or circuit are combined in adifferent manner and/or replaced or supplemented by other components ortheir equivalents. Accordingly, other implementations are within thescope of the following claims.

What is claimed is:
 1. A portable communication device comprising: acolor image sensor to generate a first image signal having a color imagesignal; a monochrome image sensor to generate a second image signalhaving a monochrome image signal; and one or more processor adapted to:obtain the first image signal and the second image signal respectivelyfrom the color image sensor and the monochrome image sensor, wherein thefirst image signal and the second image signal are generatedsubstantially at a same time, generate an image by composing a brightcomponent of the first image signal and the second image signal andadding a color component of the first image signal.